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TECH^CAI FiFTn 

The instant mventian comprises a metfaod. process or algorithm, and variations thereon, which method 
mchides motwn detection, crosssiissolvi,* and shape interpolation; devices or ^ms for practicing that method 
and. product (genenUly motion picmre fihn, videotape or videodisc, anatog or digitt^^ 
magnenc or optical media, or a transmission, taoadcast or other dism^ 



25 art, 



SCOPE OF IKVmnCWI XKn PPinP APX 

TTie bstant invendon comprises a method, process or algoridmi. and variations Uiereon. including motion 
detection, crossniissolving and shape imerpobtion: devices or systems for pracddqg that mediod; and. produa 
(generaUy motion picmie fihn. videotape or videodisc, analog or digitaUy stored motion sequences on magnetic or 
opncal media, or a transmission, broadcast or odier distribution of same) produced by the mediod and/or system 

llie purpose to 1^ the invention is appBed is to process (generaUy by digital computt^ 
a motion picmre sequence in order to produce a processed motion picmre sequence which exhibits: ^ 
die perceived quality of flutt sequence when viewed; and/or a decrease of the requirements fo^ 
or transmission resources without signiftcantly effectmg image quality (i.e.. data compression or bandwiddi 
reduction). 

hi order to acconqilisb diese benefits. Inventor will be relying on a mimber of methods and devices diat are 
weU-known. well-developed. weD-documented and witiiin die ken of intended practitioners and diose skilled in die 



The intended practitioner of the present mvention is someone who is skilled in designing, implementing, 
integrating, bmlding. creating, programming or utiliziqg processes, devices, systems and products, such as dios^ 
diat encode a higher-definition television or video signal imo a lower-defmition television or video signal suitable 
for transmission, display or recording; record, transmit, decode or display such an encoded signal; transduce or 
inmsfa an image stream from an imaging elemem to a transmission or storage element, such as a television camera 
or fihn chain; transfer an hm«e stream fiom a agnal mput to a recording medhmi. such « 
recorder; transfer an nnage stream ftom a recording medhmi to a dispby demem. such as a video 
player; transfer data representing images from a computer memory element to a display elemem. such as a 
framestore or frame buffer; syndiesize an image output stream from a madiematical model, such as a computer 
graphic rendering component; modily or combme image streams, such as image processing components, time-base 
correctors, signal processing components, or special effects components; products dial result from the foregoing- 
and many odier devices, processes and producu diat M wiUtin die reahns of motion picture and televisioil 
engineering, or computer grqihics and hnage piocessuig. 

That is. one skiUed in ±e an required to practice die instam mvemion is capable of one or more of 
following: design and/or construction f devices, systems, hardware and software fi.e.. programmiitg) for motion 

pictore and television production, motion picmie and television post produ^ 

computer graphics, and die like. Tliat is. motion picmre and television ei«^ 

and prognmimers. n««e processntg sysnai designers and prtigrammers. digital » 
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coaununicaiion and iofonnadon processing cngineeis. appUed m«hp^.t;>^^n; etc. 

Tliose skilled in die art knw how u, acconiplish such tasks as 10^ 
HnegiBie sjweins. design softwaie for and progtm tf«»e devices and s^ 

to create information p»duct. «*id. devices and ^ transfer and/or tnn«form infonnation derived ftom image 
streams. Further, such practitioners aie skilled m pnmding -sofn^ 

algonthms. programs, utilities, subroutines and libraries and to take the output from one such program and direcl 
u to the input of another . Sometimes that task requires that the output data 
Its use as iipit. ami s,«h ffle and data converdon is also within the skfl^ 
dev^ and qrstans comprise *en blown digitals 
detads of accomplishing such standard Odes are 

(mand ofthemselves) within thescopeoflheh^amimmlonialfli^ 
uses and new designs are. These known eh»«^ 
oisclosuie. 

Rather, what wiU be disclosed are novel and high-level: image analysis and prooessbg algorithms- 
iBfoiaaiHmflowsjaad. w<MgiB.Disdesedwinbewhatffl^ 

M«h Which he is aheady femlliar . m Older m implemem .he insumt invention. TT«se algoriUm« and system designs 

be pnsented by description, algetaaicfiHrnnlaeandgrap^ 
of motion pictore and television engineering, image pncessmg and computer graphics.' 
. 'I^d«criptiom.formulaeaadilh«arations 

can be miplememed in a straightforward mamu:r by programming a prognumnable computer imag^ 
as a frame buffer. 

F<>'««nple.ttepiogiammableftameb«fte (some with onboard s^^ 
graphics and/or signal processn^) suiBile for use with personal «avu^ 

along with off-the-shelf assemblen. compilers, subroutine libraries, or utilities, routinely provide as standan! 
f eamres. capabilities which permit a user to (among other tasks) : digitize a ftame of a video signal in many different 
formats inchiding higher-tium-television resotations. standard television resolutions, and lower-tium-television 
resotations. and at 8- 16- 24- ««1 32-bits per pi«l; display a video signal in any of those same formats change 
unto prognmi oomrol. the resotation ami/or bit-depth of the digitized or displayed fhm«; transfer infonnation 
between any of a) visible framestoie memory, b) blind (non-dispbyed) framestore memory, and c) host computer 
memory, and d) mass storage (e.g.. magnetic disk) memory, on a pixel-by-pixel. line-by-line. or rectai«le-by- 
rectangle basis.^ 

^ Thus, off-du^shelf devices provide die eml user with die ability to: digitize high- or low-resolution video 
ft™: access the individual pixels of those ftames: manipul^ 

host computer control ami processmg. to create arbitiarily processed pixels: ami, display processed frames, suitable 
for recordmg. comprisu« tfiose processed pixeb. niese off-the-shelf capabilities are suffidem to implement an 
image processing system embodying the information manipulation algoritimis or system designs specified herein 

Smularly. higher performance and throughput (as weU as higher cost and more programmiiig effort) 
programmable devices, suitable for broadcast or ti««arical production tasks, provide similar and much more 
sophisticated capabilities, inctodiag microK»diag whereby image processiug algorithms can be incorporated mto 
general puipose hardware, are available as off-ifae-shelf programmable systems.* 

Additionally, specialized (graphic ami image processii«) progiammable microprocessors are available for 
mcorporari "i^o^Pt^ hardware capable of providing spedaHmipose or ge^^ 
image mampulation functions.' 

I^. it is weU known by ti««« skilled m the art how to adapt processes to 
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software luimiag on programmable hardware devices, to designs for special purpose hardware, which may then 
provide advantages in cost vs. performance. 

In smnmary. the disclosure of die insiam invention wiU focus on what is new and novd a^ 
die details of vibat is known in the art. 

One of die major applications intended for die instam invention is die incorporation of die aigoriduns 
disdosed herem into a fihn chain (a fihn to video transfer device). Such transfers 

of die television motion picmre indasoy. Much time and effoit is expended in achieving desired and artistic resulB. 
And, in particular, the scene-by-scene color correction of such transfen is common practice. 

Hms. in die instam disclosure, it wiU be suggested diat practitioners make adjusmients to die operational 
parameteis of die disclosed algoiiflims in order to better achieve desired resute 

practitioners diat such individual adjustments may be appUed to images or image portions exhibiting different 
characteristics. 

Inventor's earlier relevant and published woik. inchides die following: 

1. Early work in fihn colorization lead to die developmem of using shape interpolation (sometimes caUed 
hnage warpmg) and cross^iissoMng. as apidied to key-frame color signals, for die reduction of information 
storage and processing re^iirements. 

2. Later work in fihn colorization and 2D to 3D conversion comprised, in part, improved mediods of 
generating image boundary information. 

3. Later work in 2D to 3D image conversion comprised, in pan. die creation of 3D images by: extracting 
texnire maps and 3D shape and motion information from motion picture sequences; and. re-applying diose 
textures to odier versions <rf die 3D ahqies widi whidi diey were originaUy associated widi. 

4. Work in image conqwession and bandwidfli reduction lead to die tovekqimem of pioc^ 
time-varying data selection and arrangement (widi improved percepnial results): off-lme compwation and 
recording for bandwidtii reduction; variable pixel geometry; and. die incorporation of additional 
information into die blanking intervals of a frame prior to die one widi which diat additional information 
is to be associated at recqwion. permitting multi-frame-time and/or pipelined decoding and reintegration 
of diat additional information. 

5. A version of Inventor's paper, SlBtEOSfnmssis: A Process for Adapting Traditional Media far 
Stereognphic Displays and Virtual Reality Environments, Proceedings of The Second Annual Conference 
on Vimial Reality. Artificial Reality, and Cyberspace, San Francisco, Meckler. 1991. provides further 
details on his StereoSynthesis'" 2D to 3D image conversion technology. 

The foUowiog are publicly available, in die prior an. not (in and of diemselves) die subject of die instam 
uwention, and widun die knowledge and familiarity of diose sUUed in the art.* 

1 . Shape and Morion from Image Streams under Orthography: a Factorization Method, Carlo Tomasi and 
Takeo Kanade. International Journal of Computer Virion, vohune 9, number 2, pages 137-154. Ktaiwer 
Academic Publishers. The Nedierlands 1992. 

2. Shqte and Motion frtm Image Streams: a Factorization Mexhod-Pan 3: Deteaion and Tracing of Pmm 
Features, Carlo Tomasi and Takeo Kanade, Carnegie MeUon University. Pittsburgh 1991. 

3. The Magic of Image Processing (Ouster ft Morphing), Mike Morrison. SAMS Publishing. Indianapolis 
40 1993. 

4. Four papers from: Computer GrapMcs: Proceedings afOie 1992 SIG<atAPH Cm^rence; V hune 26. 
Number 2. July 1992, ACM Press, New York 1992. 

a. Fem«reflajed/ina|ge«07>/i6?g,ThaddeusBder and Shawn Necly, at page 35. 

b. Scheduled Fourier Volume Morphing, John F. Hughs, at page 43. 
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c. yi^sicalfy Based Approach to 2.D Shape Blending, Tbo^ 
Greentvood, at page 25. 

d. ShapeTnmsfbmoiionforPofyhedmlObjectt^ 
Parcm,atpagc47. 

5. ^^k'^PonemRecognaionandlnu.gePnKessingiC^^ 

Vjymg Images; Ouster 14 Determining Three.&nensicnal Motion and Struaure from Two Perspective 

au9ter9bnageSegmemadon),M.lzzyY. Young. Acaifemic P«ss. Inc.. New York 1986 

iirf^nnr """^ 2D and 3D shape and motion 

infonnanon fiom motion sequences; and. tbe dettcdoa. creation and use of unage boundaries and segments 

«^°°^»>'«^*'*i«and.lau^.cdor,elevia 
^^sten.ofier5251inefian«.30.^ 

625 hue frame but run at a frame rate of 25 per second. Higher resolution military and laboratory video systems 
r:eL"i;r^ ^O^^V,^^ ^^P- . improve 

'-^""^.-"^"Pic-^fitoisprojec^dW 
twrce.Rec«dy. a system was devdopedl^Dougto ^ 
picnires each shown only once, to improve visual quaUty. ' 

When color was added to US black & white television, it was decided to adopt a -compatible" system 
^l^black^^tesetstotec^ivecolort^onsignalsand^^^ 

bystan*«I.elev«on.dBplayi,gs«md«^ 

more v,deo hn« and more pixeb (from WCmn. El^: vh^ 

the standard u.lev«ion chamiels can be used to transmit a -compatible" sumdard resototion signal while a second 
diannel (noueceivable by a standard television) be used to transmit the -inbetween" higher resotation information 
However. fflXW may also display a wider picmre when compared with standard television. Inclusion of the extra 
sidestnps ma compatible broadcast system has been one of the main problems. 

^»»«^««»>«dpractioetotiansmitmotionpicmiefihn.whfc^ 
ftame nue. over a broadcast television chamud by use of a fi^ 
toatdevision camera, dxefihn chain synchronizes d« .wo ta^ 

cmera has been replaced by a digital image sensor and digital frame store. ^ 
of two mterieaved video fidds. resdting In 60 fields per second. US fi^ 

ma ratto of 2 5 ^ fields per fihn frame. PracticaUy. this is achieved by alternating 3 repeated video fields and 
2 rq^ated vuleo fields for alternate fihn frames. Tie ^ 
Of the Video camera. 

^»^^««^««^Pn«i««OB«enneq^ 

devices such as character (text) generators, computer graphic systems and specu^ 

fiill-frame stores and comb filters. ^ 

•n«««^«nanytechniqttesforbandwu^ 
^liedtotelev.aon«o«.ems.Tl«searepart«^ 
tnnsmissi n. «4iere resources are limited. 



DESCRIPTinK OF INVRNTinM 



The instam invemioa conqmses a method, process or algorithm, and 
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detection, cross-dissolving and shape interpolation; devices or systems for pracdcing dial mediod: and. product 
(generally motion picnirc film, videotape or videodisc, analog r digiiaUy stored motion sequences on magnetic or 
optical media, or a transmission, broadcast r other distribution of same) produced by the method and/or system. 

TTie puipose to ^Osidx the invention is apphed is to process (generaUy by digital computer image processing) 
a motion picmre sequence m order to produce a processed motion picmre sequence which exhibits: an increase in 
die perceived quaUty of dial sequence when viewed; and/or a decrease of die requirements for information storage 
or transmission resources without significantiy effecting image quaUiy (i.e., data compression or bandwiddi 
reduction). 

In order to miderstand die mvention more fully, it is helpful to examine certain aspects of fdm and video 
display systems, flieir shortcomings, and die functioning of die human visual system. The reader is directed to 
consult die parent qqilication, of which die instant application is a continuation-in-pan, for furdier details. 

Spatial/Temporal Characteristtcs of Film and Video .Sv5rrPMg> 

Fdm and video display systems each have dieir own characteristic "signamre" scheme for presenting visual 
mformation to die viewer over time and space. Each spatial/temporal signamre (STS) is recognizable, even if 
sublinunally, to die viewer and contributes to die identifiable look and *%el* of each medium. 

Theatrical fihn presentations consist of 24 different picnires each second. Each picmre is shown twice to 
increase die -fUcker rate** above die direshold of major annoyance. However, Vfhen objects move quickly, or 
contrast gready, a phenomenon known as strobing happens. The viewer is able to perceive diat die motion sequence 
is actually made up of individual pictures and motion appears jericy. This havens because die STS of cinema 
cameras and projectors is to capnire or display an entire picmre m an instam, and to miss all die information diat 
haiqiens between these instants. 

In dnematogr^y, die proportion of time die shutter is open during eadi l/24di second can be adjusted. 
Keeping die shutter open for a relatively long time will cause moving objects to blur. In "stop motion" model 
photogr^hy it is now common practice to leave die shutter open while die model is moved for each ejqx>sure, ratiier 
dian to take a series of static images (die technique, first popularized at Industrial light and Magic, is referred to 
as *go motion" photography). In both cases, eadi motion picmre frame is taken over a "long" instam, while objects 
move. This does cause motion blurring, but does also lessen die perception of strobing; die ""smttering" namre of 
" die fihn STS has been lessened by temporal smearing. 

A phenomenon related to strobing, ^ch also is more noticeable for contrasty or fast moving situations, 
is call doubling. As noted, each motion picmre frame is shown twice to increase die flicker rate. Thus, an object 
shown at position A in projected ifirame 1 , would again be shown at position A in projected frame 2, and would 
finally move to position B in projected frame 3. The human eye/brain system (sometimes called die Retinex, for 
RETinal-oerebral con^lEX) expects die object to be at an mtermediate position, between A and B, for die 
intermediate frame 2. Since die object is still at position A at fiame 2, it is perceived as a second object or ghost 
lagging behind die firtt; hence, doubling. Again, dus is a consequence of die STS of fihn projection. The overall 
result is a perceived jitteriness and muddiness to motion picmre film presentations, even if each individual picmre 
is crisp and sharp. 

Video, on die odier hand, works quite differendy. An electron beam travels across die camera or picmre 
mbe , tracing out a raster pattern of tines, left-to-rigfat, top-to-bottom, 60 times each second. The beam is mmed off, 
or blanked, after each line, and after each picmre, to allow it to be repositioned widiout bemg seen. 

Except for die relatively short blankmg miervals, television systems gadier and display mformation 
continuously, ahhough, ai any given time, mformation is being displayed for only one *'point" on die screen. This 
STS ism marked comrast to diat of fihn. Some defects f su^ a system are diat die individual lines (or evm dots) 
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crfften^ttrpanemniaybeseeobecausetiu^isonlyalimit^^ 
faUScann»«cialtelevisions^^ 

t«n^ ^ on fton. to boaom. Howev^. ^ 
at a different tune, diere is no sense of doubling. 

nresen.r«r.^ w "^'^ ^ *^ ^'^^ *e repeated 

presematen of identical infonnadon to die human visual system niis can d«nnn«r«<.H k u 
transfer«.yi . V. . demonstrated by watching material 

fflm to video usmg newer equipment As explained above, each film ftame is repeated for ei*er 
3«2v,deofie.^*^^er.Ne^fihnchah«canpan.p„nord.tacro^ 
dc^ng so, each video fidd cotuains unique infotmadon. 

a differem position or oriemation. During diose brief sequences when a c«^ 

equipmem,d«re is a perceivable increased clarity to die scene. «iea by the film cham 

In smmnan.. film systems deal widi infonmoion everywhere at once, but for only small slices of time 
Te^on ^ ^ ^ (3^, 3„ ^ ^ 

which J^TZT^"" "° "^'"^ "^"^ """"^ ^^^"^ u-bniques to create an STC scheme 
^d. wdl reduce bod. t,^ of perceivable anomalies and 

transmit image motion sequence signals. 
THElNVKNTinwmitaupp- 

The basis of die instant invemion is d«t die hmnan visual system responds better to information display 
sysuans duu presem unique information at each frame. Standard dieatrical motion piaure fihns provide only 24 
umqt^ii^of 48 presemedeach second. On die odu^hand, st^ 

P.^d« 60 mnque field imag« each secoml, but at lower resohition; and. Showscan provides bodi high temporal 
and high geometric lesohnion. «»"i~nu 

TTie instam invention will employ high-level algoridm« ««I system designs to process motion picmte 
sequences (originating in fihn. video or odierwise) m produce fihn. video or digital presemations dutt meet d« 
umquene^req^ Tins ^ be done by syndiesizing information frames for times intennediate to dmse 
avadabh^Ttalower-levdalgbridmisim^olvedm^ ^ • 

»^on and crossHlissolvmg. Tl« h»t two. in combination, are sometimes referred to as "transition'^ 

««m by tte apphcation of much computation and. optionaDy. some human imetvemioa and assistance. The Zs 
can be recorded (perhaps, in an ofiF-line mamier) and d«n distnlmted via any stam^ 

or as any sumtod mformation product. In particuh.. die processing of images derived from standard dicatricai 
mo,Kmpjcm«fihnat24FPSmproducevW^ 

fa^^tion. sinceahigher-ftamendeanage stream can be cre«^ 
some embodmiems Will permit a reduced-frame rate image s.^^^ 

additional mouon specification information, and a higher-frame rate hnage stream con«nu«d at dJrecepdL (or 

acce^and display site. Ihus. a data compression or bandwiddi 

may be used to reduce storage or transmission requiremems. or can be used t^ 
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to the image stream whidi can comprise: additional resolution or definition: additional image area (e.g., wide-screen 
side-strqjs): 3D infoimaiion in the fonn of a second image, or from which two images can be created by combination 
wifliflie first; interaaive r game data; hyper- or multimedia data; image segmentation data 
or where differem algoridmis are to be applied; or, the imerleaving f several program channels. 

In particular, it is noted thai, in addition to standard television broadcasting, such compression is verj' 
desirable for a number of odier applications. Specifically: so-called «500 channeP cable (or via satellite broadcast, 
fiber or phone line) television; digital image streams to be displayed firan conqniter disk or CI>-ROM: image streams 
via communication lines for on-hne multimedia or video conferencing; storage of video signals on analog or digital 
tape (or other magnetic or optical media); the transmission of HDTV, stereographic television, or new -digital" 
television sigmds. 

pHTAH Fl> nRSCRlPnON W ITH DRAW^KHS 

What foUows is a detailed description with drawings that will iUustrate several preferred embodiments of 
the instant invention. 

Referring, first, to Table I, below, note that: fihn firame 0 exacUy corre^nds in time with an even video 
field 0; film fi:ame 1 Ms between even video field 2 and odd video field 3; 

fihn frame 2 exactly rortespcmds in time with an odd ^ field 7 and 

even video field 8; and, fihn frame 4 exactly corresponds m time wiA an cvM 10. starting the repeat of 

the l/6th second temporal cycle. 



1/120 SECOND CLICKS 
VIDEO FIELD TYPE 
VIDEO FIELD COUNT 
FILM FRAME COUNT 

TABLE 1: TemnnTal Alignmeni Film Frai^iHt and Viden FieWj^ 

In the tersest terms, the basic embodiment of the invention wiU be to use shape interpolation and cross- 
dissolving (i.e., a process akin to image morphmg) to derive, fiom pairs of film images, intermediate images, for 
the purpose of presenting unique and temporally appropriate images at each video field. 

Table U shows the setting of the morph parameter (0% to 100%) and vrtiich fihn unages are used to create 
each video field. Note that a moxph parameter of 100% corresponds to using the first of the two fihn frames alone 
and uiyrocessed. Similarly a morph parameter of 0% would correspond (if used) to using the second of die two fihn 
frames alone and unprocessed TTie number m parenthesis is die complementary percentage from die perspective of 
die second frame. 

F i rst F i lm Framp Sgcgnd Film Trmf^ ^Momh' Parr.n^ot-f^r video Fi^id 

100% (0%) 0 e 

60% (40%) 1 o 

20% (80%) 2 e 

80% (20%) 3 o 

40% (60%) 4 e 

100% (0%) 5 o 

60% (40%) 6 e 



e o 
0 1 
0 



e o 
2 3 
1 



e 
4 



o 
5 
2 



e 
6 



o e 
7 8 
3 



o 
9 



e 
10 
4 



0 
0 
0 
1 
1 
2 
2 



1 
1 
1 
2 
2 
3 
3 
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^ 3 20% (80%) 

I * 80% (20%) 

* 40% (60%) 



3 

4 5 



7 o 

8 e 

9 o 



100* { 0*.) 10 e temporal 

repeat 



here 



TABLE n: Mrnnhmg fenm.^.^ for win. , ^ vm^^ Pi^|^^ 



AS dKmn. Ihe image data is derived from fihn feunes. For ime^^lation. shape data is also required 
TTus may be provided by a compuier/tamaB coUabonuive system such as that disclosed by Inventor for film 
cotorization or 2D to 3D image conversion (or as used IV TOO. Ito«e «fcr tt^ 

patems and applications for qrstemdiagnuns; only the ^ 
^.bsequently disdosed by Invemor, such systems can also be made to 

the mcoiponmon into the system of additional software capabilities to extract image boundary (segmentation) 

information and/or motion data. Similariy. ft^ 

Out may be used to inqilement die moiphing functions. 

Such automatic operation v« considered less than q«imal In^ 
necessary to identify and separate acmal objects from within the frame. At 1^ j, 
« only necessary to identify the areas of the image that move furespective of w^ 
world coherem objects) or which need be associated from key frame to key frame. 

om. film frame and the next (within a scene) are gemaally quite small. In contrast. Inventor's fihn colorization 
system employed key frames many fihn frames .pan. Therefbie. the use of automatic boundary extracdon 
(paniculariy based on motion) and motion analysis algorithms wiU provi^ 
close in time "micro-morphing" task at ^^p^ 

In particular, a technique that extracts -optical flow- wiU be used as follow^^ 
mformauon. what is extracted is a field showing how the various areas (e.g.. individual pixels) of the unage are 
movmg (both magnitude and direction) from frame to frame. This information may inchide translation sizing 
skewmg or rotation chaises. See Figure 1. Additionally, pixels may "appear" or -disappear" as object routes and 
new areas come from behind or old areas go out of view. Similarly, as objects mumaHy intersect, portions may 
become newly visible or obscured. See Figure 2. 

Tliis optical flow data can be used m Ueu of the imerpolated boumlaries to pi^ 
a morphiug like fimction. with an imerpolated fidd function apph^ 

Inparttcular.«viicalfloworothermotionda«amaybepioWdedovertheentireimage^^ 

on a regular grid). See Rgurc 3. TT« data can then be imetpolated between those poa^^ 

appropriate values for eat* pixd in the hnage. For embodiments ^ this data win have to be ti^^ 

below) data may be sem only for certain of the poum m each frame. Those poims with the m«^ 

be sem.oramo,e regular paismg may be emptoyed.Fbrasimpleexampk. if one considetsachft^^ 

on such a grid, the -black poims- may be alternated with the -white poims\ At each frame fl^ 

current set will be given heavy weight; however, the poims sem for prior or subsequem frames may ato 
consulted (pethaps avenq ed over time) but. perhaps with less weight. Alternately, a more complex "variable STC" 

W« of pattern may be empl<ved to select which points to tr^ 

name. 

Whichever technique is employed for nnage warping, the percentages ofTaWenarea^ 
as weU as the cross^lving fimction. and unique frames are created for each vi^ 
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20 



frames). 

The above will be accomplished either auiomancally or with human operator participation; but, in many 
embodimmts (particularly «*ere optical flow compulations are being used lo compute motion for image warping) 
the process wiU be accomplished in an off-line manner. Thai is. the image analysis and processing computations will 
be done on a frame-by-frame basis (although, particularly for the analysis, several frames will be "considered" 
sinmltaneously) and these frames win be created, coUected and committ^ 
time basis. 

For other embodiments, the motion/change/shape data calculations wiU be performed but, rather than 
producing die new frames, die old frames and die motion data will be recorded or transmitted. Upon access or 
recqjtion, die low-frame rate image data and modon data will be combined, in real time, to create a full-frame rate 
image stream. Hie advem of veiy-hi^-perftmnance consumer electronics (e.g., interactive game settop boxes and 
die like) will provide a hardware environment widiin which such computations may be carried out. See Figure 4. 
Pipelined architecmre and variable geometry frame stores (as disclosed in Inventor's other appUcauons) will be 
useful to implOTient such devices. Further, for such real-time applications, computationaUy simpler embodiments 
15 will be preferred. 

EvamiaUy. settop boxes and die like, may become available \n*ich can, in real time, perform die entire 
process (motion analysis and morphing). Until that time, botti image and motion data will have to be delivered and 
utilized. Several embodiments of how to acconqilish diis foUow. 

In a straightforward embodiment, image data frames may be ahemated widi sh^ or motion data. And diat 
shape or motion data may be associated widi die previous image data, die later image data or "inbetween the two**. 
See Figure 5. 

If shqie data are used, the shapes are interpolated between shape data frames. 

If motion data are used, die motion of&ets may be ^lied in several ways. If a motion offset data frame 
is supplied, it can represem a l/120di second change. Urns, for a video field at or after the time of die fifan image: 
25 for a 100% morph parameter die offset is not ^lied since die image is used unchanged; for an 80% parameter it 
is applied once; for a 60% parameter it is applied twice (in succession or twice as strongly); for a 40% parameter 
it is applied diree times; for a 20% parameter it is ^lied four times; for a 0 % parameter it is not applied since die 
image is not used. 

Sunilarly, for a video field at or before die time of die fihn image: for a (100%) marft parameter die offset 
30 is not applied since die image is used undianged; for a (80%) parameter it is applied once but widi a reversed sign; 
for a (60%) parameter it is applied twice (in succession or twice as strongly) but widi a reversed sign; for a (40%) 
parameter it is applied diree times but with a reversed sign; for a (20%) parameter it is ^plied four times but widi 
a reversed sign; for a (0%) parameter it is not applied since the image is not used. 

Alternately, die shape or motion frame may be considered to be ''between" die image frames. Then die 
35 same shape/motion data firame will be applied to die image frames on eidier side, but in opposite directions. If an 
image frame is die first of die pair die shape/motion frame to die right is applied widi positive sign; if an fanage 
frame is die second of a pair, die shape/motion frame to die left is applied with negative sign. See Figure 6. 

>^idi eidier sh2^ interpolation or motion offset application, if only two shape or motion data frames are 
applied a linear interpolation between the two is posable. 

However, for more sophistication, the values from one or more frames before and or after die frame (or 
frame pair) in question can be consulted. Thus, curve fitting algorithms (e.g., splines) can be applied to all data 
dimensions (translations in X and Y, rotations, skews, size chaiiges. sources or smks; or more wdi 3D sh^ie/motion 
data). In diis way. more naniral and sophisticated diangw, diat progress non-linearly, from frame to frame, can be 
computed. See Figure 7 for exanoples shown for a single parameter. 
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J" TkTT "^"^ -y be sto^d as. or sem via. video ^.h some addiuonal iafonnation 

rif' """^ ^'•'^ used «, hold nvo film images, nvo fields mav be applied .0 

each film ftaH«. wid, ti« iwo shapcto«ion daia fa««s comained in fifth field. However. ±. shape/motion dau, 
can. m^ be put in fl« blaBld«g in^rvak of th^ 

Z^^'T"? " ' ^- Rn^. by applying line doubling inten^lauon 

^ can be «>len«ed smce fun-frame video provides much beoer respond 

««meeachofthetwoftamesneedbesent.andthenihieeoffi^^ SeeRgureS 

as much as 60% Of ti« image stream) may be used for: addiuonal resolution or 
defimnon Cm both d«cti«»s or in bit-depth); additional image area (e.g.. HDTV, wide-screen or "letterbox" side- 
smps); 3D Mornudon m the fonn of a second image, or from which two images can be created by combination 
w,&±efi^ mteracnveorgamedanuhw-r-ormultim^^ 

or where drffcrem algorithms are to be applied; or. the imerleaving of several program chamteU. THe specifics of 
Aese will not be disclosed here, some have already been disclosed by Inventor in other applications or patents 
Tte details of such use. in general, are no. in and of themselves considered the subsumce of the p«sent invemion 
(e«:ept where specific novel details axe provide*; however, the application of the -morphing" fhm»e creation 
process and the ensuing -freei,g „p- of video bandwidfli. nsdting m tiiese possible uses, is tite substance of the 
piesem mvention. . wcwuic 

As explained, above, system diagnmB for the instam invemion are vi^ 
toventor m earlier applications, for either computer assisted or automatic systems. However, an information «r 
software flow diagram is provided as Figure 9. 

Next, a more sophisticated embodimem is described, which wiU be particularly useful where pixel sinks 
and souths occur, ami which was also described in 
Virtual Reality* presematioiis based on films. 
In this case: 

1. I°^e analysis algorithms are first apphed to ttie image sequence to extnua 3D shape «^ 

2. Tlie tamups representing the surfeces of these objects are extracted from d>e image and ti,e inverse of ti» 

inigection transform is used 10 -miwrap- the surfiu* images from die 3D shapes derived m 

texmre maps for each 3D object. TTmse may be pieced together from several images eitiier 
stream of the frame in question. 

3. Based on flie 3D motion data extracted in step l.imennediate 3D frame scenes are created 
or resbi^ing each 3D object. 

4. J^'«chol>ject.texmremapsfromsourceimages.oneitiiersideoftheimermediateftam^ 
are crossslissolved (or the closest texnue m^ may be used). 

5. •n«««=««nKvsarelha,reappiiedio.hedistorted«^^ 
stereoscopic pairs of 2D projections) are created as intermediate frames. 
See Figure 10. 

'"'^^bove may be used as an alternative to the 2D embodimemswWch came before, or aspe^ 
embodm«m may be combined. It is less likdy tiu« tins 3D embodimem WiU 

feshron. and less likely stiU dat it may be used for a real-time system (a least witi, currem commercial level 
technotegy). Nevertheless, for processi,* 24 FKti^atrical motion pic^ 
tnmsfer. these .echni.p«s may be usefal m process p«blem«ic sce^ 

Thesetechniquesmaybecombnu^lwitholhcrdatarednctiontechniq^ ^ 
^e segmemation data described elsewhere, dam may be sem/stoml. in add^ 

frames, so tiiat various areas of frames in a sequence may be assembled frmn several methods 
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For example (see Figure 11): 

1 . Some areas may be reoincd firom one frame to the next. Particularly since analysis of moiion data will be 
an inqxmant aspect of die basic instant invention areas thai lack mouon or change will be detected. Thus, 
pan of the data sent can inchide (or be deduced from the motion data sent) a map of area that move so litUe 
that then need not be updated for at least the currem frame. 

2. Some areas may change so drastically that the presem inveuion will not prove adequate and, for those areas 
(also indicated by some {presumably highly conQ)rKsed} area map) replacement data would be sent which 
may be compressed by any conq)atible data compression technique now extant or later developed. 

3. Tbos& areas remaining may be mterpolated by the techniques disclosed herein. 
The flows depicted in the software flow diagrams herein are exemplary, some items may be ordered 

differently, combined in a single stqp. skipped entirely, or acconvlished in a different manner. However, the 
depicted flows will work. In particular, some of these functions may be carried out by hardware components, or by 
software routines residing on, or siq)plied with, such a componeiu. 

Similariy die systems ducted in tixe system diagrams herein are exemplaiy, some items may be organized 
differemly, combined in a single dement, split into multiple elements, omitted entirely, or organized in a different 
manner. However, die dq>icted systems wffl work. In particular, some of diese fimctions may be carried out by 
hardware components, or by software routines residuqg on, or sqiplied widi, sucb a component. 

It wiU dius te seen dxat die objects set fordi above, among diose made ^iparent from die preceding 
description, are efiBcientiy attained and certain changes may be made in carrying out dte above mediod and in die 
consmictira set fordi. Accordingly, it is intended diat all matter contained in die above description or shown in die 
accompanying figures shall be interpreted as iUustrative and not in a limiting sense. 

While diere has been shown and described what are considered to be preferred raibodiments of die 
invention, it wOl, of course, be understood diat various modifications and changes in form or detail could readily 
be made widiout departing from die spirit of die invention. It is, diercfore, intended diat die invention be not limited 
25 to die exact form and detail herein shown and described, nor to anydiing less dian the whole of die invention herein 
disclosed as hereinafter claimed. 
I claim: 
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NOTKS 

1. Typicai examples include: 

Digital Video: Selections from the SMPTE Journal and Other PubUcations. Sociei>' of Moiion 
Kcmze and Televisios Engineers, Inc. (SMPTE). 1977. 

Digital Video Volume 2. SMPTE 1979. 

Digital Video Volume 3, SMPTE 1980. 

Gnphics Engines, Margciy Conner. Hectronic Design News (EDN), Cahners Publishing Company. 
Newton. MA, Vohnne 32, Number 5. March 4, 1987, pages 1 12-122. 

Algorithms for Cnphics and Image Processing, Theo P^vlidis. Computer Science Press 1982. 

Conqtuter Vision. Ballard and Brown, Prentice-Hall, Englewood Cliffs 1982. 

Industrial AppUcmions ofMaOme Vision, IEEE Conqmter Society, Los Angeles 1982. 

Structured Computer Vision, Ed. Tanimoto and Klinger, Academic Press, New York 1980. 

Computer Architecture for Pattern Analysis and Image Database Management, IEEE Computer 
Sodety Press, Hot Springs 198L 

Image Procesmg System Ardiitectures, Kittler & Duff, John Wiley & Sons, Inc., New York 1985. 

Multiresolunon Image Processing and Analysis, Ed. A. Rc^enfeld. Springer-Verlag. New Yoik 
1984. 

Image Reconstruction from Projections, Gabor T. Herman, Academic Press 1980. 

Basic Methods of Tomography and Inverse Problems, Langenberg and Sabatier, Adam Hilger, 
Philadelphia 1987. 

US Patent Number 2,940,005 issued June 7. 1960. Inventor: P. M. G. Toulon. 

Principles (tf Interactive Computer Graphics, Second Ed. , Newman & SprouU, McGraw-HiU Bo(^ 
Company, New York 1979. 

Advances in Image Procesmg and Pattern Recognition, Elsevier Sdenoe Publishers B.V.. 
Amsterdam, 1986. 

Image Recovery Theory and ^plication, Hemy Stark, Academic Press, Inc., New York 1987. 

Handbook of Pattern Recognition and Image Processing, Ed. Tzay Y. Young, Academic Press 
Inc.. New York 1986. 
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Fuiidamemals qflnteracdve Compmer Graphics, Foley and Van Dam, Addison-Wesley New York 
1982. 

Real Linear Algebra, Anatal E. Fekete. Maxcel Dekkcr. Inc., New York 1985. 

Fame Dimermonal MuUilinear Algebra, Parts l&Il, Marvin Marcus, Marcel Dekker, Inc New 
York 1973. 

Sparse Matrix Compuumons, Ed. Bunch & Rose, Academic Press, Inc., New York 1976. 

Matrix Confutations and Mathematical Software, John R. Rice. McGraw-HiU Book Company New 
York 1981. 

The Architecture of Pipelined Con^uters, Pfeter M. Kogge, McGraw-HiU Book Company New 
York 1981. 

Digital System Design and Microprocessors. John P. Hayes, McGraw-HiU Book Conmany New 
York 1984. 

Digital Fibers and the Fast Fourier Transform, Ed. Bede liu, Dowden, Hutchenson and Ross. Inc.. 
Stroudsburg 1975. 

Hardware and Software Concepts in VLSI, Ed. Guy Rabbai, Van Nostrand Reinhold Company. Inc., 
New York 1983. 

Digital Signal Processing, Qiqpenheim and Schafer, Prentice Hall. Inc., Englewood Cliffs 1975. 
Movements of the Eyes, R. H. S. Carpenter. Pion, Limited, London 1977. 
Service Manual: DCX-SOX) 3-Chip CCD Video Camera, SONY Corporation. 
Color Television: Principles and Servicing 1973. 

Multi-Dimensional Sub-Band Coding: Some Theory and AlgoriOu^ 

Processing 6 (1984) 97-112, Hvsevier Science Publishers B.V. North-Holland, p. 97-112. 

The Laplacian Pyramid as a Compaa image Code, Bun and Adelson, IEEE Transactions on 
Conununications, Vol. Com-31, No, April 1983, p. 532-540. 

Exaa Reconstruction Techniques for Tree-Structured Subband Coders, Smith & Barnwell, IEEE 
ransactions on Acoustics, Speech and Signal Processing, Vol. ASSP-34. No. 3 June 1986, p. 434-441. 

Theory and Design ofM-Oumnel Maximally Decimated Quadrature Mirror FUters mth Arbitrary M, 
Having the Perfea Reconstruction Property, P.P. Vaidyanatixan, IEEE Transactions on Acoustics, 
Speech and Signal Processing. V 1. ASSP-35, No. 4, April 1987, p. 476-492. 

Application of Quadrature Mirror FUters to ^lit Band Voice Coding Schemes, Esteban & Galand, 
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IBM Labonuory. 06610, La Gaude, Fiance. 

f^^on Television wiih High Piaure QualiO', Bnxlcr Wendiand. SMPTE Journal 
October 1983. p. 1Q28-103S. 



2. See, for exanqile: 

wf^I'f^f''''^'^^"'**'^'*'^'^^^^ Society Of Mouon 

Picnire and Television &igineers. Inc. (SMPTE), 1977. 

Digiud Video Volume 2, SMPTE 1979. 
IXgital Video Volume 3, SMPTE 1980. 

^en^ Definition Television yviA High Piaure Qualiiy, Bioder Wendiand. SMPTC Journal 
October 1983. p. 1028-1035. ^ricjoumai, 

T^TrZT^' "f"^ ^'^^ Coherence-, Volu»e 26. Number 2. July 

1992, ACM Press, New York 1992. ^ 



3. For«anq>le: 

PIP-512, PIP-1024 and PIP-EZ (software): PG-640 & PG-1280; MVP-AT & taager-AT (software) 
aUforitemM-PC/AT,ftom Matrox Electronic Systems, Ltd. (3ue., Canada. '''^^~"*»">' 

T^^er Graphics Series (hardware and software), for the IBM-PC/AT. from Pixelworks. New 

TARGA (several models with software utUities) and AT-VISTA (with software available from the 
manufecmrer and Texas Instruments, namifacmrer of the TMS34010 onboard Graphics System 
P«>«««»'«*ip).f« the BM-PC/AT.ftom AT&T Epicenters 



TTie tow-end Pepper Series and high-end Pepper Pn, Series of boards (with NNIOS software . and 
mcludmg the T«as Instruments TMS34010 onboard Graphics System Processor chip) from Number 
Nine Computer Corporation, Massachusetts. 



4. For example: 

ros^ and FGS^SOO high-resohition imaging systems fiom Broadcast Television Systems, 



91 1 tophics Engine and 91 1 Software Library (that runs on an IBM-PC/AT comiected by an 
mteifece cord) from Megatek. Corporation, c»\itnnvn 
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Oiie/80 and One/380 frame buffers (wiib software from manufacouer and third panies) from Raster 
Technologies, Inc., Massachusetts. 

Image processing sysienis manufacmied by Pixar, Inc.. California. 

And many different modehs of graphicHapable workstations from companies such as SUN and 
Silicon Graphics, Inc.. including the Indy, Indigo and ONYX series. 



5. For Example: 

GMP VLSI Gr^cs Microprocessor from Xtar Electronics. Inc., Illinois. 

Advanced Graphics Chip Set (inchiding the RBG. BPU, VCG and VSR) from National 
Semiconductor Corporation, California. 

TMS34010 Graphics System Processor (with availabte Software Development Board, Assembly 
Language Tools, '•C" Cross-Compiler and other software) from Texas Instruments, Texas, 

6. Other useful references include, for exanq)le: 

The Inserpreuuion of Visual Motion, Ulhnan. MIT Press, Cambridge 1992. 

Processing Differemial Image Motion, Rieger and Lawton, Journal Optical Society of America. Vol 
2. No. 2, February 1985. 

(hi the Si0ciency of the Velocity Field for Perc^tion of Heading, Warren. Blackwell, Kurtz, 
Hatsopoulos and Kalish, from Biological Cybernetics, Springer-Verlag 1991. 

Numerical Shape from Shading and Occluding Boundaries, Ikeuchi and Horn, Artificial InteUigenoe 
17, North-Holland Publishing Company 1981. 

Processing Translational Motion Sequences. Lawton, Con^uter Vision Graphics and Image 
Processing 22. Academic Press, Inc. 1981. 

The Imerpretanon of a Moving Retinal Image, Longuet-Higgins and Prazdny, Proceedings of the 
Royal Society of London 1980. 

Objea Recognition by Affine Invariant Matching, Lamdan. Schwartt and Wolfson, IEEE 1982. 

Sight and Mind, Kaufman, Oxford Press, New York 1974. 

Perception: An AppUed Approach, Schiff, Copley Publishing Group, Acton 1990. 
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Ao^auKi for achieving incnasedvis^ 

» a second image sequence of a second, higher, frame «ie by applying shape imenK,laUon and c««s- 

dissolvmg tt, pairs of source images u, create imennediam images and coii* 
image sequence widi said intennediate images. 



2. Amethodasinclaiml.«bereinsaidfir8tim^,equenceis«24fta,teperseco^ 
sequence is at 60 frames (fidds) per second. 



3. A product created by the process of claim 2 and recorded on film. 

A product created by the process of claim 2 and recorded on videotape. 
5. A product created by the process of claim 2 and distributed via an information^ 



i. A process for data reduction whereby an image sequence to be displayed at a second, higher, frame rate 

« ston«l or tiansmitUHl as an image sequem* of a first, lower, ftame nue imerspers^ 
data frames. 



A process for image display of the data reduced image sequence of claim 6 whereby saM 

of a first, lower, ftame rate, is processed, along with said imerspersed shape/motion data to produce 

intermediate ftames which are displayed in combination with said first image sequence. 

An impnnred process for image compressiM whereby, for each Win an image disp^^ Uu, 

next miage in said image display sequence is constructed by some combination of : reu^ 

of the prev«,us fiame in said image sequence: replacing some portion of the p«^ 

data for said next frame; and. creating, by shape imetpolation between said previous ftame and a 

subsequent frame, some portion of said next frame. 

A p«KhHn created by comreying on an information beaiiflg medha. 

of claim 6. k'«»*«> 
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(901) SHAPE /BOUNDARY DATA CREATED BY IMAGE ANALYSIS OR WITH 
HUMAN/MACHINE COLLABORATION 



(902) PROPORTIONALLY WEIGHTED (BASED ON TIME POSITION OF IMAGE) 
DISTORTION IS PERFORMED ON TWO IMAGES ON* EITHER SIDE OF 
INTERMEDIATE FRAME TO BE CREATED 



(903) PROPORTIONALLY WEIGHTED (BASED ON TIME POSITION OF IMAGE) 
CROSS-DISSOLVE BETWEEN TWO DISTORTED IMAGES 



FIGURE 9: Software Flow Diagram of 2D InterpQlat ion Embodiment 



I (1001) IMAGE ANALYSIS ALGORITHMS EXTRACT 3D SHAPE AND MOTION |[ 



(1002) BITMAPS EXTRACTED FROM OBJECTS AMD ""UNWRAPPED" BY 
APPLYING INVERSE PROJECTIVE GEOMETRY TRANSFORMS 



(1003) 3D SHAPE AND MOTION DATA USED TO PRODUCE INTERMEDIATE 
3D SCENES" WITH REPOSITIONED/RESHAPED OBJECTS 



1 (1004) TEXTURE MAPS SELECTED OR 


CROSS-DISSOLVED BETWEEN 1 






1 (1005) TEXTURE MAPS REAPPLIED TO NEWLY CREATED 3D OBJECTS | 



FIGURE 10: Software Flow Diagram of 3D Intemol ation Embodiment 
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